Skip to content

Persist real face embeddings during actor indexing (unblocks #74) - #151

Open
ik020 wants to merge 1 commit into
grayhatdevelopers:mainfrom
ik020:fix/actor-index-persist-real-embeddings
Open

Persist real face embeddings during actor indexing (unblocks #74)#151
ik020 wants to merge 1 commit into
grayhatdevelopers:mainfrom
ik020:fix/actor-index-persist-real-embeddings

Conversation

@ik020

@ik020 ik020 commented Sep 5, 2026

Copy link
Copy Markdown

Prep Work for #74 — Find People Using a Reference Image

The Bug

Actor indexing detects a face, aligns it, and computes a normalized SFace encoding for each detection — but that encoding was never actually persisted.

_actor_records() was writing a placeholder embedding=[0.0] into every StorageRecord sent to Chroma. As a result, the actor collection's vector index has never contained real, searchable face embeddings.

During indexing, matching only used ActorIndexState.known_encodings, which is an in-memory list scoped to a single indexing run and discarded once the run finishes.

This silently blocks #74: finding a person using a reference image requires comparing a new face embedding against all previously detected faces across the repository, but there was no durable embedding data to compare against.

The Fix

  • Carry the computed encoding through into each detection dictionary in process_actor_samples.
  • Update _actor_records() to persist the real, normalized encoding as StorageRecord.embedding instead of [0.0].
  • Leave _actor_cluster_records() unchanged. A cluster summary spans multiple detections and does not represent a single face image, so it has no single embedding to persist.

Schema Version Bump

Because this changes what is actually persisted for each actor detection, existing generations with the actor modality enabled contain placeholder vectors and are no longer trustworthy for face search.

The repository already has a mechanism for handling this:

  • CompletedGenerationManifest.index_schema_version is typed as Literal[INDEX_SCHEMA_VERSION].
  • local_snapshots.py::validate_generation() validates the manifest on disk using model_validate_json().
  • ValidationError is explicitly caught and re-raised as IndexSchemaError.

This PR bumps INDEX_SCHEMA_VERSION from 7 → 8.

As a result, old generations will fail validation cleanly with IndexSchemaError instead of silently being treated as complete, searchable indexes containing meaningless vectors.

User-facing impact

Any existing generation with the actor modality enabled will need to be re-indexed after this change.

This is intentional. The alternative would be allowing #74 to search against invalid placeholder embeddings with no indication that the underlying data is unusable.

Testing

New test

Added:

test_actor_indexing_persists_real_face_embeddings_not_placeholders

This drives process_actor_samples through mocked YuNet/SFace calls end-to-end and verifies that the StorageRecord sent to storage.upsert contains the real normalized encoding rather than the placeholder [0.0].

Updated test

Updated:

test_actor_records_preserve_stable_detection_metadata

The test previously created fake detection dictionaries without an encoding field. It now provides an explicit encoding and verifies that the resulting StorageRecord.embedding matches the real value.

Test results

Full test suite:

590 passed, 2 skipped, 0 failed

Also re-ran the tests covering INDEX_SCHEMA_VERSION and actor indexing:

  • test_generation_manifest.py
  • test_local_snapshots.py
  • test_indexing.py
  • test_actor_results.py
  • test_cli.py
  • test_frontend_app.py

All tests pass, including the existing test that constructs a manifest with INDEX_SCHEMA_VERSION + 1 to verify that the rejection path still works.

Not Included

This PR does not implement reference-image search itself.

Reference-image search remains the actual feature tracked by #74 and will be implemented as a separate follow-up PR once this change lands, since it depends on real actor embeddings being persisted first.

 prep)

Actor indexing computed a normalized face encoding per detection but
never stored it: _actor_records() wrote a placeholder embedding=[0.0]
into every StorageRecord, so the actor Chroma collection's vector
index has never contained anything searchable. Matching only ever
happened in-memory, within a single indexing run, via
ActorIndexState.known_encodings, and was discarded once that run
finished.

This blocks issue grayhatdevelopers#74 (find people using a reference image), which
requires comparing a reference image's embedding against previously
indexed faces across the whole repository.

Fix:
- Carry the computed encoding through into each detection dict.
- _actor_records() now writes the real normalized encoding as the
  StorageRecord embedding instead of the placeholder.
- Cluster-summary records (_actor_cluster_records) are left
  unchanged: a summary rolls up multiple detections and has no
  single face image to encode.

Because this changes what's actually stored per detection, any
existing generation with the actor modality enabled has placeholder
vectors and is no longer valid. Bump INDEX_SCHEMA_VERSION 7 -> 8 so
CompletedGenerationManifest's Literal[INDEX_SCHEMA_VERSION] check
rejects old generations with a clear IndexSchemaError instead of
silently treating placeholder-vector indexes as complete and
searchable. Affected generations need to be re-indexed.

Adds a new end-to-end test driving process_actor_samples through
mocked detector/recognizer calls, asserting the stored embedding is
the real normalized encoding rather than [0.0]. Updates the existing
_actor_records test, which built detection dicts without the
now-required encoding key.
@SaadBazaz

Copy link
Copy Markdown
Member

Hey @ik020,
Thanks for the Pull Request.
As a starter, we require all contributors to "star" and "fork" the repo, in order to determine if there really is a human behind the wheel or an autonomous agent.
Please star the repo for our review to proceed.
Thanks!

@ik020

ik020 commented Sep 7, 2026

Copy link
Copy Markdown
Author

Hey @ik020, Thanks for the Pull Request. As a starter, we require all contributors to "star" and "fork" the repo, in order to determine if there really is a human behind the wheel or an autonomous agent. Please star the repo for our review to proceed. Thanks!

Thanks! I’ve already forked the repo and have been working on my branch. I’ve spent the past week working on it and have fixed 4 issues with PRs submitted.

Just to clarify, I’m not an autonomous agent 😄,I’m a final-year CS undergraduate at FAST NUCES Islamabad. Our university shared this opportunity with us via email, which is how I found the repo and started contributing.

I’ll star the repo as well. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants